[megatron, model] feat: qwen3.5 example by ISEEKYAN · Pull Request #5381 · verl-project/verl

ISEEKYAN · 2026-02-24T06:53:36Z

What does this PR do?

thanks to @LiuXTao 's great work on ISEEKYAN/mbridge#83, the mbridge has supported qwen3.5.

This PR succeeded in running qwen3.5 SFT on verl based on mbridge supports for qwen3.5

Checklist Before Starting

Search for similar PRs. Paste at least one query link here: ...
Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
- {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward
- If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
- {type} is in feat, fix, refactor, chore, test
- If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
- Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

see examples/sft/gsm8k/run_qwen3_5_megatron.sh
and examples/grpo_trainer/run_qwen3_5-35b-megatron.sh

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

Read the Contribute Guide.
Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
Add / Update the documentation.
Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)
If your PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main.

gemini-code-assist

Code Review

This pull request adds support for Qwen3.5 SFT with Megatron. The changes are mostly workarounds and fixes to support the Qwen3.5 architecture, particularly its Gated Delta Net (GDN) and chat template requirements. The changes look reasonable and well-commented, improving compatibility and robustness. I have one major concern about catching a broad Exception which could hide bugs.

gemini-code-assist · 2026-02-24T06:56:14Z

verl/utils/dataset/multiturn_sft_dataset.py

+                return_tensors="pt",
+                **apply_chat_template_kwargs,
+            )
+        except (jinja2.exceptions.TemplateError, Exception) as e:


Catching a generic Exception is risky as it can suppress unexpected errors, making debugging difficult. It's better to catch more specific exceptions. Since jinja2.exceptions.TemplateError is a subclass of Exception, the tuple (jinja2.exceptions.TemplateError, Exception) is redundant and equivalent to except Exception:. Please replace Exception with the specific exception type(s) that are expected to contain the 'No user query' message. If the exact type is unknown, consider catching a narrower set of exceptions like ValueError or TypeError which are common for such issues.

wuyaoxuehun · 2026-02-25T06:56:14Z

@ISEEKYAN does this pr can also support rl?

ISEEKYAN · 2026-02-26T03:38:07Z

@ISEEKYAN does this pr can also support rl?

just updated a script with RL supports. But it is not easy to prepare a right vllm dependency now🥲

wuyaoxuehun · 2026-02-26T13:59:56Z

@ISEEKYAN does this pr can also support rl?

just updated a script with RL supports. But it is not easy to prepare a right vllm dependency now🥲

Many thanks.
"I succeeded in running this script with the main branch of vllm on 20260225, yet there are still some minor issues

the vllm qwen3.5 during initialization, need to be fixed."

so what issue is there with vllm qwen3.5 initialization?

I see in vllm doc that vllm can indeed serve qwen3.5(https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3.5.html).

khazic · 2026-02-27T07:56:13Z

Successfully ran Qwen3.5 SFT (verl megatron example) with the following setup:

(1) mbridge: install from source for qwen3_5 support —
pip install git+https://github.com/ISEEKYAN/mbridge.git
(PyPI 0.15.1 does not register qwen3_5; use git to get qwen3_5.)

(2) megatron-core == 0.16.0 — required for attention_output_gate and other GDN options.

(3) verl patch in verl/models/mcore/patch.py: applies the gate-slicing fix when
num_query_groups < tp_size (same as Megatron-LM PR #3529), plus the existing
mbridge compatibility patch. Without the gate patch, training fails with a
shape mismatch in _apply_output_gate.

Key library versions used:

mbridge: from git (reports 0.15.1)
megatron-core: 0.16.0
transformers: 5.2.0
torch: 2.9.0+cu129
flash_attn: 2.8.1
flash-linear-attention: 0.4.1

verl/workers/engine_workers.py

wuxibin89 · 2026-03-03T05:28:07Z

examples/grpo_trainer/run_qwen3_5-35b-megatron.sh

+#   Qwen3.5 uses Gated Delta Net (GDN) linear attention which currently does
+#   NOT support packed sequences (THD format) in Megatron-LM. Therefore:
+#     - actor.megatron.use_remove_padding=False  (forces bshd compute format)
+#     - model.use_remove_padding=True           (keeps NestedTensor in data pipeline)


For new model engine, I think we always use NestedTensor regardless of model.use_remove_padding?

wuxibin89 · 2026-03-03T05:39:24Z

verl/utils/tensordict_utils.py

+        # Try the fast path first: direct unbind works for some NestedTensor
+        # layouts where the batch dim is not entangled with the ragged dim.
+        try:
+            tensors = nt.unbind(dim=0)


In which case nested tensor unbind failed? I didn't expect that unbind may failed.

3D jagged tensors (e.g., MRoPE position_ids)

verl/utils/dataset/multiturn_sft_dataset.py

FlyingDutchman26 · 2026-03-03T10:37:57Z

It seems that the vllm nightly version which support Qwen3.5 require transformers==4.57.6, which conflicts with the latest transformers version 5.2.0

ISEEKYAN · 2026-03-03T14:40:15Z

It seems that the vllm nightly version which support Qwen3.5 require transformers==4.57.6, which conflicts with the latest transformers version 5.2.0

you could try with sglang

zyfzjsc988 · 2026-03-04T02:55:07Z

It seems that the vllm nightly version which support Qwen3.5 require transformers==4.57.6, which conflicts with the latest transformers version 5.2.0

try first install vllm==0.15.0 then install transformers==5.2.0

zyfzjsc988 · 2026-03-04T03:07:17Z

Successfully ran Qwen3.5 MoE GRPO (this PR example) with the following setup:
vllm: 0.15.0
mbridge: git main branch (Commit 3dd5af9)
megatron-core: 0.16.0
transformers: 5.2.0
torch: 2.9.0+cu129
flash_attn: 2.8.1
flash-linear-attention: 0.4.1

however, when i ran Qwen3.5-27B, error raise:

  File "/root/verl/verl/workers/megatron_workers.py", line 887, in compute_log_prob
    output, entropys, layers_topk_idx = self.actor.compute_log_prob(data=data, calculate_entropy=not is_lora)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/utils/profiler/performance.py", line 105, in f
    return self.log(decorated_function, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/utils/profiler/performance.py", line 118, in log
    output = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/workers/actor/megatron_actor.py", line 254, in compute_log_prob
    output = self.forward_backward_batch(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/workers/actor/megatron_actor.py", line 716, in forward_backward_batch
    losses_reduced = forward_backward_func(
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/pipeline_parallel/schedules.py", line 636, in forward_backward_no_pipelining
    output_tensor, num_tokens = forward_step(
                                ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/pipeline_parallel/schedules.py", line 423, in forward_step
    output_tensor, loss_func = forward_step_func(data_iterator, model)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/workers/actor/megatron_actor.py", line 666, in forward_step
    output = forward_fn(
             ^^^^^^^^^^^
  File "/root/verl/verl/models/mcore/model_forward.py", line 141, in model_forward
    output_orig = model(
                  ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/distributed/data_parallel_base.py", line 22, in forward
    return self.module(*inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/module.py", line 489, in forward
    outputs = self.module(*inputs, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mbridge/models/qwen3_5/model.py", line 367, in forward
    output = self.language_model(
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/models/gpt/gpt_model.py", line 504, in forward
    preproc_output = self._preprocess(
                     ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/models/gpt/gpt_model.py", line 388, in _preprocess
    rotary_pos_emb = self.rotary_pos_emb(
                     ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mbridge/models/qwen3_vl/rope_utils.py", line 130, in forward
    seq_expanded = seq[:, :, None, :].float()
                   ~~~^^^^^^^^^^^^^^^
IndexError: too many indices for tensor of dimension 2

@ISEEKYAN is this PR support Qwen3.5-27B? or any example for Qwen3.5-27B?

ISEEKYAN · 2026-03-04T06:45:37Z

@ISEEKYAN is this PR support Qwen3.5-27B? or any example for Qwen3.5-27B?

@zyfzjsc988 I only did exp on moe version, let me check if there is any bug with dense version. cc @LiuXTao

FlyingDutchman26 · 2026-03-04T07:16:06Z

Successfully ran Qwen3.5 MoE GRPO (this PR example) with the following setup: vllm: 0.15.0 mbridge: git main branch (Commit 3dd5af9) megatron-core: 0.16.0 transformers: 5.2.0 torch: 2.9.0+cu129 flash_attn: 2.8.1 flash-linear-attention: 0.4.1

@zyfzjsc988 Hi, thanks for sharing your setup and experience with the Qwen3.5 MoE GRPO example!

I tried to reproduce your environment and also installed transformer-engine, but I'm still encountering an error:

ValueError: Model architectures ['Qwen3_5MoeForConditionalGeneration'] are not supported for now

I suspect that vllm==0.15.0 might not be the correct version to support this model architecture. Shall we install the nightly version or if there are any additional steps needed to make it work?

Any guidance would be greatly appreciated. Thanks in advance!

zyfzjsc988 · 2026-03-04T07:36:17Z

@ISEEKYAN is this PR support Qwen3.5-27B? or any example for Qwen3.5-27B?

@zyfzjsc988 I only did exp on moe version, let me check if there is any bug with dense version. cc @LiuXTao

hi, @ISEEKYAN i fix this bug and run Qwen3.5-0.8B (Qwen3.5-27B TP4 also works) just now by adding QWEN3_5_VL config to SupportedVLM, and you can also check if there are others needed to modify for dense model.

class SupportedVLM(Enum):
    QWEN2_5_VL = "Qwen2_5_VLForConditionalGeneration"
    QWEN3_MOE_VL = "Qwen3VLMoeForConditionalGeneration"
    QWEN3_VL = "Qwen3VLForConditionalGeneration"
    QWEN3_5_MOE_VL = "Qwen3_5MoeForConditionalGeneration"
    QWEN3_5_VL = "Qwen3_5ForConditionalGeneration"

but Qwen3.5-27B still have bugs when TP=8, and i am not sure if mbridge needs modify, for example:

  File "/root/verl/verl/workers/megatron_workers.py", line 887, in compute_log_prob
    output, entropys, layers_topk_idx = self.actor.compute_log_prob(data=data, calculate_entropy=not is_lora)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/utils/profiler/performance.py", line 105, in f
    return self.log(decorated_function, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/utils/profiler/performance.py", line 118, in log
    output = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/workers/actor/megatron_actor.py", line 254, in compute_log_prob
    output = self.forward_backward_batch(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/workers/actor/megatron_actor.py", line 716, in forward_backward_batch
    losses_reduced = forward_backward_func(
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/pipeline_parallel/schedules.py", line 636, in forward_backward_no_pipelining
    output_tensor, num_tokens = forward_step(
                                ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/pipeline_parallel/schedules.py", line 423, in forward_step
    output_tensor, loss_func = forward_step_func(data_iterator, model)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/verl/verl/workers/actor/megatron_actor.py", line 666, in forward_step
    output = forward_fn(
             ^^^^^^^^^^^
  File "/root/verl/verl/models/mcore/model_forward.py", line 141, in model_forward
    output_orig = model(
                  ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/distributed/data_parallel_base.py", line 22, in forward
    return self.module(*inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/module.py", line 489, in forward
    outputs = self.module(*inputs, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mbridge/models/qwen3_5/model.py", line 367, in forward
    output = self.language_model(
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/models/gpt/gpt_model.py", line 525, in forward
    hidden_states = self.decoder(
                    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/transformer_block.py", line 619, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/module.py", line 352, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/transformer_block.py", line 765, in forward
    hidden_states, context = layer(
                             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/transformer_layer.py", line 1217, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/module.py", line 352, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/transformer_layer.py", line 513, in forward
    hidden_states, context = self._forward_attention(*args, **kwargs)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/transformer_layer.py", line 597, in _forward_attention
    attention_output_with_bias = self.self_attention(
                                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mbridge/models/qwen3_5/attention.py", line 360, in forward
    core_attn_out = self._apply_output_gate(core_attn_out, gate)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/megatron/core/transformer/attention.py", line 1221, in _apply_output_gate
    gate = gate.view(*x.shape)
           ^^^^^^^^^^^^^^^^^^^
RuntimeError: shape '[776, 1, 768]' is invalid for input of size 1191936

zyfzjsc988 · 2026-03-04T07:49:58Z

Successfully ran Qwen3.5 MoE GRPO (this PR example) with the following setup: vllm: 0.15.0 mbridge: git main branch (Commit 3dd5af9) megatron-core: 0.16.0 transformers: 5.2.0 torch: 2.9.0+cu129 flash_attn: 2.8.1 flash-linear-attention: 0.4.1

@zyfzjsc988 Hi, thanks for sharing your setup and experience with the Qwen3.5 MoE GRPO example!

I tried to reproduce your environment and also installed transformer-engine, but I'm still encountering an error:
ValueError: Model architectures ['Qwen3_5MoeForConditionalGeneration'] are not supported for now
I suspect that vllm==0.15.0 might not be the correct version to support this model architecture. Shall we install the nightly version or if there are any additional steps needed to make it work?

Any guidance would be greatly appreciated. Thanks in advance!

please try install vllm v0.16.1rc0

FlyingDutchman26 · 2026-03-04T10:31:00Z

@zyfzjsc988 @ISEEKYAN Thank you so much for your previous help. I've successfully run the GRPO training script.

However, I encountered an issue with transformers==5.2.0:

  File ".../lib/python3.12/site-packages/transformers/modeling_rope_utils.py", line 651, in convert_rope_params_to_dict
    ignore_keys_at_rope_validation = ignore_keys_at_rope_validation | {"partial_rotary_factor"}
                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for |: 'list' and 'set'

As a temporary workaround, I modified the transformers source code locally:

if ignore_keys_at_rope_validation is None:
    ignore_keys_at_rope_validation = set()
elif not isinstance(ignore_keys_at_rope_validation, set):
    ignore_keys_at_rope_validation = set(ignore_keys_at_rope_validation)

After this change, the training ran successfully.

I was wondering if you have encountered the same issue, or if there's a more proper fix (e.g., updating to a newer transformers version)?

Thanks again for your guidance!

zhaichenhao1889 · 2026-03-06T08:17:31Z

when i use qwen3.5 9b the follow error happen

ray.exceptions.RayTaskError(NameError): ray::WorkerDict.ref_init_model() (pid=507880, ip=10.141.0.100, actor_id=1ef0b433cb57b0ae2fef392701000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x15247d7a7130>)
File "/scratch/zhaizhongyang/.conda/envs/work_verl/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/scratch/zhaizhongyang/.conda/envs/work_verl/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/scratch/zhaizhongyang/code/work_verl/verl/verl/single_controller/ray/base.py", line 932, in func
return getattr(self.worker_dict[key], name)(*args, **kwargs)
File "/scratch/zhaizhongyang/code/work_verl/verl/verl/single_controller/base/decorator.py", line 427, in inner
return func(*args, **kwargs)
File "/scratch/zhaizhongyang/code/work_verl/verl/verl/workers/megatron_workers.py", line 625, in init_model
self.ref_module, self.ref_model_config = self._build_model_optimizer(
File "/scratch/zhaizhongyang/code/work_verl/verl/verl/workers/megatron_workers.py", line 453, in _build_model_optimizer
ref_module, updated_tf_config = make_megatron_module(
File "/scratch/zhaizhongyang/code/work_verl/verl/verl/utils/megatron_utils.py", line 279, in make_megatron_module
model = bridge.get_model(
File "/scratch/zhaizhongyang/.conda/envs/work_verl/lib/python3.10/site-packages/mbridge/core/bridge.py", line 123, in get_model
model = get_model(
File "/scratch/zhaizhongyang/.conda/envs/work_verl/lib/python3.10/site-packages/mbridge/core/util.py", line 112, in get_model
model = build_model()
File "/scratch/zhaizhongyang/.conda/envs/work_verl/lib/python3.10/site-packages/mbridge/core/util.py", line 102, in build_model
model = model_provider_func(
File "/scratch/zhaizhongyang/.conda/envs/work_verl/lib/python3.10/site-packages/mbridge/models/qwen3_5/base_bridge.py", line 814, in provider
transformer_layer_spec = self._get_transformer_layer_spec(vp_stage)
File "/scratch/zhaizhongyang/.conda/envs/work_verl/lib/python3.10/site-packages/mbridge/models/qwen3_5/base_bridge.py", line 57, in _get_transformer_layer_spec
get_transformer_block_with_experimental_attention_variant_spec(
File "/scratch/zhaizhongyang/.conda/envs/work_verl/lib/python3.10/site-packages/megatron/core/models/gpt/experimental_attention_variant_module_specs.py", line 188, in get_transformer_block_with_experimental_attention_variant_spec
backend = _get_backend_spec_provider(config=config)
File "/scratch/zhaizhongyang/.conda/envs/work_verl/lib/python3.10/site-packages/megatron/core/models/gpt/experimental_attention_variant_module_specs.py", line 367, in _get_backend_spec_provider
else TESpecProvider()
NameError: name 'TESpecProvider' is not defined

baobaohanhan21 · 2026-03-06T08:31:41Z

@zyfzjsc988 @ISEEKYAN Thank you so much for your previous help. I've successfully run the GRPO training script.

However, I encountered an issue with transformers==5.2.0:
  File ".../lib/python3.12/site-packages/transformers/modeling_rope_utils.py", line 651, in convert_rope_params_to_dict
    ignore_keys_at_rope_validation = ignore_keys_at_rope_validation | {"partial_rotary_factor"}
                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for |: 'list' and 'set'
As a temporary workaround, I modified the transformers source code locally:
if ignore_keys_at_rope_validation is None:
    ignore_keys_at_rope_validation = set()
elif not isinstance(ignore_keys_at_rope_validation, set):
    ignore_keys_at_rope_validation = set(ignore_keys_at_rope_validation)
After this change, the training ran successfully.

I was wondering if you have encountered the same issue, or if there's a more proper fix (e.g., updating to a newer transformers version)?

Thanks again for your guidance!

I've also successfully run the GRPO training script. modify to set(ignore_keys_at_rope_validation) is a proper fix currently. but I encountered CPU OOM when save_checkpoints.

add qwen3.5 megatron sft example

44f5a1d

ISEEKYAN requested a review from vermouth1992 as a code owner February 24, 2026 06:53

gemini-code-assist bot reviewed Feb 24, 2026

View reviewed changes

wuxibin89 mentioned this pull request Feb 25, 2026

[roadmap] model support list tracking #5389

Open

submit an RL script

d9611d9

ISEEKYAN requested review from FightingZhen, PeterSH6, ji-huazhong and tardis-key as code owners February 26, 2026 03:36

ISEEKYAN changed the title ~~[megatron, model] feat: qwen3.5 megatron example of SFT~~ [megatron, model] feat: qwen3.5 example Feb 26, 2026

conver334 mentioned this pull request Feb 27, 2026

[megatron] feat: support Megatron-FSDP mode for Megatron backend #5423

Draft

8 tasks

wuxibin89 reviewed Mar 3, 2026

View reviewed changes

verl/workers/engine_workers.py Outdated Show resolved Hide resolved

wuxibin89 reviewed Mar 3, 2026

View reviewed changes

ISEEKYAN added 2 commits March 3, 2026 09:03

update doc of chunk_tensordict

a418672

update apply_chat_template_single_turn

80c0f87

ISEEKYAN force-pushed the mcore_qwen35 branch from 1da3a2f to 80c0f87 Compare March 3, 2026 17:47

ISEEKYAN requested a review from ArronHZG as a code owner March 3, 2026 17:47

Merge branch 'main' into mcore_qwen35

9336800

ISEEKYAN added 2 commits March 4, 2026 00:11

clean

1bf133a

clean

98571a9

small fix

28b106e

Conversation

ISEEKYAN commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist Before Starting

Test

API and Usage Example

Design & Code Changes

Checklist Before Submitting

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

wuyaoxuehun commented Feb 25, 2026

Uh oh!

ISEEKYAN commented Feb 26, 2026

Uh oh!

wuyaoxuehun commented Feb 26, 2026

the vllm qwen3.5 during initialization, need to be fixed."

Uh oh!

khazic commented Feb 27, 2026

Uh oh!

Uh oh!

wuxibin89 Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

wuxibin89 Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

ISEEKYAN Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FlyingDutchman26 commented Mar 3, 2026

Uh oh!

ISEEKYAN commented Mar 3, 2026

Uh oh!

zyfzjsc988 commented Mar 4, 2026

Uh oh!

zyfzjsc988 commented Mar 4, 2026

Uh oh!

ISEEKYAN commented Mar 4, 2026

Uh oh!

FlyingDutchman26 commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zyfzjsc988 commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zyfzjsc988 commented Mar 4, 2026

Uh oh!

FlyingDutchman26 commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhaichenhao1889 commented Mar 6, 2026

Uh oh!

baobaohanhan21 commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ISEEKYAN commented Feb 24, 2026 •

edited

Loading

FlyingDutchman26 commented Mar 4, 2026 •

edited

Loading

zyfzjsc988 commented Mar 4, 2026 •

edited

Loading

FlyingDutchman26 commented Mar 4, 2026 •

edited

Loading